Learning Temporal Sequences using Bag of Features

This paper presents a novel method for learning classes of temporal sequences using a bag-of-features approach. Bag-of-features techniques have recently been introduced with success in the computer vision community for image classification, object recognition and categorization. We transpose these techniques from the image domain to the temporal domain. More exactly, we show that such techniques can be applied to the case where inputs are temporal sequences of features representing events to classify, and therefore for tasks such as gesture and activity recognition. In our approach, a codebook of local of temporal patches, or codewords, is automatically constructed from a set of sample sequences. Temporal sequences are then encoded using accumulated histograms of parts from this dictionary. Based on this representation, a multi-class classifier, treating the bag of features as the feature vector, is applied to estimate which class to assign to the temporal sequence. Obviously, this type of approach would not be able to explicitly learn the temporal relationships between the features in the sequence. However, as our study shows, the use of overlapping descriptors evaluated at multiple temporal scales are able to implicitly learn these temporal relationships. The segmentation problem is also addressed. For each learned temporal patch, a temporal model of the sequence boundaries is learned and used to simultaneously segment and classify the temporal sequences. Finally, extensive experiments are performed on three different datasets to compare our method against state-ofthe-art algorithms, including Hidden Markov Models, Conditional Random Fields and Support Vector Machine. The results show that our algorithm performs better and requires less training data than competing algorithms.